Apparatus and method for improving, augmenting or enhancing vision

ABSTRACT

The present invention provides various apparatus and methods for improving, augmenting or enhancing a vision of a person that include a first camera, one or more sensors, a microdisplay, and one or more processors communicably coupled to the first camera, the one or more sensors and the microdisplay. The first camera is configured to acquire a first image of a scene facing away from an eye of the person. The microdisplay is configured to display a modified first image to the eye. In one operational mode, the one or more processors are configured to acquire the first image of the scene using the first camera, modify the first image based on one or more vision improvement parameters, and display the modified first image on the microdisplay to improve, augment or enhance the vision of the person.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to, and is a national phase patentapplication of International Application No. PCT/US2015/016717, filed onFeb. 19, 2015, which claims the benefit under 35 U.S.C. § 119(e) of U.S.Provisional Application No. 61/941,777 filed Feb. 19, 2014. Theforegoing applications are hereby incorporated by reference in theirentirety.

FIELD OF INVENTION

The present invention relates generally to the field of imageprocessing, and more specifically to a system and method for improving,augmenting or enhancing vision.

BACKGROUND ART

Without limiting the scope of the invention, its background is describedin connection with Age-Related Macular Degeneration (AMD). AMD is aprogressive disease with a typical onset at the age of 60 years andlater. It begins with a loss of central vision (typically in both eyes)and often progresses leading to effectively legal blindness. In the U.S.alone, over 1.6 million persons are afflicted with AMD, with greaterthan 200,000 new cases occurring yearly. Currently, there is no cure forAMD.

More specifically, AMD patients suffer from a non-correctible loss (afew angular degrees) of central vision (typically consisting of a 6 to11 degree cone angle). This is the region of vision with the highestresolution that normal-sighted persons use for reading and discerningdetail. The eye's perceptive resolution drops precipitously as the anglefrom the central axis increases.

AMD patients adapt to the loss of central vision by shifting to usetheir closest-to-center unaffected peripheral vision (e.g., “near”peripheral vision). Peripheral vision typically remains unaffected, butits resolution is significantly lower (even for normal vision persons)so that this region of vision is not useful for any detail acuity, norcan it be rendered useful with any known techniques.

Peripheral vision alone is unable to recover the visual acuity of thelost central vision (the ability to discern fine detail). The mostrelevant difference between central and peripheral vision of the humaneye is the vastly reduced spatial resolution. In theory, an objectviewed by the near peripheral vision can be made to appear with as muchdetail (visual acuity) as by the central vision, as long as adequateoptical magnification is applied. Unfortunately, as magnificationincreases, the useful field of view is proportionally reduced. Theresulting amount of scene information that can be perceived by thismagnified region is thus also reduced. For example, low vision assistivedevices' efficacy is often rated by how much these devices impactreading speed, since reading speed is highly influenced by the number ofwords presented within the useful visual field (inversely proportionalto magnification).

Magnification has been proven to be the single most useful element tohelp AMD patients. While numerous magnification devices have beenintroduced in the marketplace, all of them have shortcomings in terms ofutility within a single device to assist in a variety of everydayactivities (e.g., near and far vision, bright and indoors lightingconditions, etc.) A common complaint from low vision persons is thatthey cannot simultaneously carry and use multiple assistive devices,each for a particular task, all the time (while remaining ambulatory,performing normal life activities hands-free).

SUMMARY OF THE INVENTION

Various embodiments of the present invention provide an apparatus andmethod for improving, augmenting or enhancing the vision of a person.Most often, the device will aid visually impaired persons, with AMD inparticular, to better see using their existing remaining vision. Thedevice could also be used to improve, augment or enhance the vision of aperson having normal vision in various commercial, industrial, medical,military and technical applications.

Persons with AMD benefit from the digitally enhanced realtime imagerypresented to the aided eye, while the unaided eye remains unobstructed.This combination allows the patient to use their undamaged peripheralvision, while augmenting their central vision. The brain automaticallyselects the aided or unaided eye based on the current task. The primaryimage enhancing software functions may include, but are not limited to,adjustable magnification, auto-focus (short and long range), contrastenhancement, artificial edge enhancement, background color substitution,anti-shake stabilization, eye-tracking and automatic image shifting. Theintegration of these functions into a single, ergonomic (size, shape,weight, center of gravity, etc.), hands-free, cost effective product,with the addition of certain technical features, which help preventdizziness, headaches, binocular rivalry and other side effects typicallyassociated with head-mounted displays, make the device practical to useall day enabling the user to undertake a variety of real-life tasks.

One embodiment of the present invention provides an apparatus forimproving, augmenting or enhancing a vision of a person that includes afirst camera, one or more sensors, a microdisplay, and one or moreprocessors communicably coupled to the first camera, the one or moresensors and the microdisplay. The first camera is configured to acquirea first image of a scene facing away from an eye of the person. Themicrodisplay is configured to display a modified first image to the eye.In one operational mode, the one or more processors are configured toacquire the first image of the scene using the first camera, modify thefirst image based on one or more vision improvement parameters, anddisplay the modified first image on the microdisplay to improve, augmentor enhance the vision of the person.

Another embodiment of the present invention provides an apparatus forimproving, augmenting or enhancing a vision of a person that includes afirst camera, a second camera, one or more sensors, a microdisplay, andone or more processors communicably coupled to the first camera, thesecond camera, the one or more sensors and the microdisplay. The firstcamera is configured to acquire a first image of a scene facing awayfrom an eye of the person. The second camera is configured to acquire asecond image of the eye. The microdisplay is configured to display amodified first image to the eye. In operational mode, the one or moreprocessors are configured to acquire the first image of the scene usingthe first camera, modify the first image based on one or more visionimprovement parameters, and display the modified first image on themicrodisplay to improve, augment or enhance the vision of the person. Inone operational mode, the one or more processors are configured toacquire the first image of the scene using the first camera, acquire thesecond image of the eye using the second camera, modify the secondimage, determining an eye gaze angle based on the second image or themodified second image, modify the first image based on one or morevision improvement parameters by offsetting the first image based on theimage offset, and display the modified first image on the microdisplayto improve, augment or enhance the vision of the person.

In yet another embodiment of the present invention, a computerizedmethod for improving, augmenting or enhancing a vision of a person isprovided. An apparatus is provided proximate to an eye of the person.The apparatus includes a first camera configured to acquire a firstimage of a scene facing away from the eye, one or more sensors, amicrodisplay configured to display a modified first image to the eye,and one or more processors communicably coupled to the first camera, theone or more sensors and the microdisplay. The first image of the sceneis acquired using the first camera and the first image is modified basedon one or more vision improvement parameters using the one or moreprocessors. The modified first image is then displayed on themicrodisplay to improve, augment or enhance the vision of the person.

In addition, another embodiment of the present invention provides acomputerized method for improving, augmenting or enhancing a vision of aperson. An apparatus is provided proximate to an eye of the person. Theapparatus includes a first camera configured to acquire a first image ofa scene facing away from the eye, a second camera configured to acquirea second image of the eye, one or more sensors, a microdisplayconfigured to display a modified first image to the eye, and one or moreprocessors communicably coupled to the first camera, the second camera,the one or more sensors and the microdisplay. The first image of thescene is acquired using the first camera. The second image of the eye isacquired using the second camera and the second image is modified usingthe one or more processors. An eye gaze angle is determined based on thesecond image or the modified second image using the one or moreprocessors. The first image is modified based on one or more visionimprovement parameters by offsetting the first image based on the imageoffset using the one or more processors. The modified first image isthen displayed on the microdisplay to improve, augment or enhance thevision of the person.

The present invention is described in detail below with reference to theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Further benefits and advantages of the present invention will becomemore apparent from the following description of various embodiments thatare given by way of example with reference to the accompanying drawings:

FIG. 1 is a high-level block diagram of a neck and head-worn apparatusin accordance with one embodiment of the present invention;

FIG. 2 is a diagram of a head and neck worn apparatus mounted oneyeglasses in accordance with one embodiment of the present invention;

FIG. 3 is a front view of the head-worn apparatus mounted on eyeglassesin accordance with one embodiment of the present invention;

FIG. 4 is a back view of the head-worn apparatus mounted on eyeglassesin accordance with one embodiment of the present invention;

FIG. 5 is a perspective view of the internal electronics within thehead-worn in accordance with one embodiment of the present invention;

FIG. 6 is a side view of the internal electronics within the head-wornin accordance with one embodiment of the present invention;

FIG. 7 is a flow chart for calibrating and configuring the settings ofan apparatus in accordance with one embodiment of the present invention;

FIG. 8 is a flow chart for processing images in accordance with oneembodiment of the present invention;

FIG. 9 is a flow chart for automatically focusing an image in accordancewith one embodiment of the present invention;

FIG. 10 is a flow chart for determining an eye gaze angle image offsetin accordance with one embodiment of the present invention;

FIG. 11 is a flow chart for motion reduction in accordance with oneembodiment of the present invention;

FIG. 12 is a flow chart for eye gaze angle gesture recognition inaccordance with one embodiment of the present invention;

FIG. 13 is a flow chart for eye gaze angle region of interest imageprocessing in accordance with one embodiment of the present invention;

FIGS. 14A and 14B are a diagram and flow chart for eye gaze angle imageoffset determination in accordance with one embodiment of the presentinvention;

FIGS. 15A and 15B are a diagram and flow chart for image scrolling inaccordance with one embodiment of the present invention;

FIGS. 16A and 16B are diagrams illustrating magnification of a partialfield of view in accordance with one embodiment of the presentinvention;

FIGS. 17A and 17B are a diagram and flowchart illustrating colorsubstitution in accordance with one embodiment of the present invention;

FIGS. 18A and 18B are diagrams and FIG. 18C is a flowchart illustratingmotion stabilization and anti-shake in accordance with one embodiment ofthe present invention;

FIGS. 19A and 19B are diagrams and FIG. 19C is a flowchart illustratingchanging magnification based on object distance and text size inaccordance with one embodiment of the present invention;

FIGS. 20A and 20B are a diagram and a flowchart illustrating wirelessimage and settings transmission in accordance with one embodiment of thepresent invention; and

FIG. 21 is a high-level block diagram of the power and data transferbetween the neck and head-worn apparatus in accordance with oneembodiment of the present invention.

DESCRIPTION OF THE INVENTION

While the making and using of various embodiments of the presentinvention are discussed in detail below, it should be appreciated thatthe present invention provides many applicable inventive concepts thatcan be embodied in a wide variety of specific contexts. The specificembodiments discussed herein are merely illustrative of specific ways tomake and use the invention and do not delimit the scope of theinvention.

To facilitate the understanding of this invention, a number of terms aredefined below. Terms defined herein have meanings as commonly understoodby a person of ordinary skill in the areas relevant to the presentinvention. Terms such as “a”, “an” and “the” are not intended to referto only a singular entity, but include the general class of which aspecific example may be used for illustration. The terminology herein isused to describe specific embodiments of the invention, but their usagedoes not delimit the invention, except as outlined in the claims.

Various embodiments of the present invention provide an apparatus andmethod for improving, augmenting or enhancing the vision of a person.Most often, the device will aid visually impaired persons, with AMD inparticular, to better see using their existing remaining vision. Thedevice could also be used to improve, augment or enhance the vision of aperson having normal vision in various commercial, industrial, medical,military and technical applications.

Persons with AMD benefit from the digitally enhanced realtime imagerypresented to the aided eye, while the unaided eye remains unobstructed.This combination allows the patient to use their undamaged peripheralvision, while augmenting their central vision. The brain automaticallyselects the aided or unaided eye based on the current task. The primaryimage enhancing software functions may include, but are not limited to,adjustable magnification, auto-focus (short and long range), contrastenhancement, artificial edge enhancement, background color substitution,anti-shake stabilization, eye-tracking and automatic image shifting. Theintegration of these functions into a single, ergonomic (size, shape,weight, center of gravity, etc.), hands-free, cost effective product,with the addition of certain technical features, which help preventdizziness, headaches, binocular rivalry and other side effects typicallyassociated with head-mounted displays, make the device practical to useall day enabling the user to undertake a variety of real-life tasks.

Various embodiments of the present invention provide devices and methodsfor improving, augmenting or enhancing the vision of persons sufferingfrom various low vision conditions, such as Age-Related MacularDegeneration (AMD). One embodiment of the device consists of aspectacle-mounted monocular electronic camera and display system. Thecamera captures images of the scene in front of the person and presentsthese to the aided eye after digital image manipulation, which mayinclude magnification, contrast enhancement, edge sharpening, etc. Thisenhances visibility with imperceptible time lag, resulting insignificantly improved visual perception under varying scene conditions(indoors and outdoors, near and distance gaze). The device is preferablyis small and light allowing it to be mounted on prescription (ornon-prescription) glasses, sunglasses, spectacles, monocles, etc. Themost common use is as a monocular (single-eye) configuration, but it canalso be used in a binocular configuration.

With respect to assisting persons with AMD, the primary function of thedevice is to magnify the images of the scene facing the user (personwearing the device), to enhance contrast, and to artificially enhanceedges (such as doorways, stairs, etc.). In order to achieve practicalfunctionality and utility for all-day usage by the user, other functionscan be included in the device, including auto-focus, auto-brightness andwhite balance, eye tracking (described later), anti-shake imagestabilization, simple and mostly automatic device operation andcontrols, in addition to long battery life. Note that the primaryfunction can be changed or altered to meet the specific vision needs ofthe person.

One embodiment of the present invention will now be described in moredetail in reference to FIGS. 1-21. The present invention is not limitedto this embodiment as it is provided for illustrative purposes only.

FIG. 1 is a high-level block diagram describing components of theelectronic device worn on the head or glasses, referred to as the HeadMounted Display Unit (hereinafter, HMDU) and the neck worn battery pack(hereinafter, the battery pack). The HMDU includes a variable focus lensfacing the scene in front of the person wearing the HMDU (hereinafter,the user) (2), a fixed focus lens facing the eye of the wearer (3), afront facing camera (hereinafter, the front camera) which capturesimages of the scene in front of the wearer (hereinafter, the scene) (4),an eye imaging camera which captures images of the user's eye (5)(hereinafter, Eye Tracking camera, or ET camera), a temperature sensor(6), a motion detector (7) comprising one or more sensors (such a 3-axissolid-state accelerometer integrated circuit) which detects the user'smotion of the head or body, an ambient light sensor (8) which detectsthe ambient light in the scene, and non-volatile user memory (9) whereall user configurations, statistics, and preferences are stored. Thesecomponents are all interfaced to a digital image processor or processors(1), such as one or more, or a combination of microprocessors,Field-Programmable Gate Arrays (FPGA), Application Specific IntegratedCircuits (ASIC) or even embedded or partially embedded within either thefront camera or the microdisplay components (hereinafter, theprocessor). The outputs of this processor are configured to send imagedata to a microdisplay (10). The microdisplay contains optics (11)enabling the eye to see the image emanating from the microdisplay. Thefront camera and display illumination (12) is controlled by theprocessor and illuminates the scene as well as the image presented tothe eye through the microdisplay.

The front camera and the ET camera face the scene and the user's eye,respectively, by one of several means or combinations of means. Thesimplest form of a camera to face its target is to be mountedmechanically in a plane that is directly orthogonal to the scene, andthat the optical path from the scene to the camera also be parallel,orthogonal and coincident with the scene. It is possible to achieve sucha final alignment of the camera to its target through other means, suchas the use of mirrors, or electronic offsets to compensate forleft-right or up-down (or even angular) misalignments, or desired imageshifts.

The battery pack includes indicator LEDs (13), user controls (14) (pushbuttons, a knob, a Cypress Semiconductor capsense electronic slider orbuttons), battery charge status monitoring (15), wireless batterycharging and power (16), USB or DC power charging and power (17), and abattery (typically rechargeable, but can also be a primarynon-rechargeable type) and battery charge circuitry (18). Amicroprocessor (19) coordinates all the elements of the battery pack andits communications to/from the HMDU. Although the current preferredembodiment is to mount the battery pack around the neck using a stylishlanyard, the battery pack may be located in other places generally onthe user's person, including behind the neck, in a pocket withinclothes, on the back of the glasses frames, on the belt, or other suchplaces.

DC power cables (20) deliver electrical energy from the battery pack tothe HMDU, and user control conductors (21) transmit the user controlcommands to the HMDU. In some embodiments, user controls may be locateddirectly on the HMDU rather than in the battery pack.

FIG. 2 shows a graphical representation of the HMDU (24) includingattachment to one lens of the eyeglasses (22) and eyeglass frames (25),and the battery pack (26). The cable (23) connecting the battery pack tothe HMDU transmits user input control data, battery status data andother data stored in the electronics worn on the neck. The battery packcontains user controls, including a capsense slider (27) or othercontrols such as a knob (28). Other controls on either unit may includebuttons, voice activated control, eye motion activated control, focuscontrol, gesture recognition control, automatic sleep/wake-up control,or a combination thereof. The controls can be remotely located withrespect to the one or more processors and other components, and maycommunicate via conductors or wirelessly. It should be noted that theHMDU does not occlude, or obscure, the peripheral vision (both to thesides or to the top or bottom) of the user's eye where the HMDU ismounted, nor does this device in the monocular (where a single HMDUused) interfere at all with the user's eye where no HMDU is mounted. Forcertain persons afflicted with vision loss such as Age-Related MacularDegeneration (AMD) the ability to derive benefit from the HMDU visionenhancements in the central portion of the user's vision whilesimultaneously not losing peripheral vision (in both the aided eye,where the HMDU is mounted) as well as in the unaided eye (the eyewithout an HMDU), is critical in enabling the beneficial use of the HMDUin all normal-life tasks.

FIG. 3 is a front view of the HMDU (38) in FIG. 2 mounted on spectacles,showing the front facing camera (35) and unobscured space to the leftand right of the eye left from the shape of the HMDU (36 and 37). Thecurrent embodiment provides for a 30 degree horizontal field of viewwhereby the user's vision is enhanced by looking at the HMDU display,while the remaining eye's field of view remains unimpeded. The HMDU canbe attached to or integrated into a monocle, pair of glasses, pair ofsunglasses, a frame or other wearable apparel to support the device. TheHMDU is typically sized to maintain a peripheral vision of the person,and is mounted or positioned in front of the eye or a pupil of the eyesuch that a second eye of the person is unobstructed.

FIG. 4 is a back view (from the user's side) of the HMDU in FIG. 2mounted on spectacles, showing the eye-box (48) where a processed imageis presented to the eye, and mounting legs (47) where the head-mounteddevice can be cemented to spectacles. The preferred mounting location ofthe HMDU is on the outside surface of the spectacles' lens, and centeredonto the user's pupil. Certain users have eccentric viewing, whereby thecenter of their pupil is not centered facing directly forwarded. Inthese cases, the HMDU may be mounted directly in front of thenon-forward pointing pupil, or an electronic image offset may beprogrammed into the HMDU to perform such an image offset to compensatefor that user's gaze eccentricity. Note that the alignment to the pupilis in both axes (up/down as well as left/right).

FIG. 5 is a view of the internal electronics and optics of the HMDU inFIG. 2, containing the necessary elements for the front camera andvariable focus lens (45) which adjusts for focus based on the HMDU'sdistance to the scene of interest, the processor and sensors (40),interconnect between eye imaging camera and one or more processors (41),optical assembly facing the eye (42, 43, 44), and eye imaging camera(46). In this embodiment, a backwards facing mirror (44) relays andmagnifies the image emanating from the microdisplay towards the user'seye. By design, the focus distance for the eye is set to long distance,which represents the most relaxed position of the rectus muscles insidethe human eye.

FIG. 6 is a side view of the internal electronics of the head-wornelectronic device, showing display illumination (51). The displayillumination uses white LEDs which can be driven to varying levels ofpower, and controlled by the processor. The level of drive of these LEDsis influenced by a number of factors, including ambient light of thescene, the sensitivity of the user's eye, and other function-specificsettings of the HMDU. Eye imaging illumination (50) consists of one ormore infrared LEDs which illuminate the eye to allow the ET camera (also50) to adequately image the user's eye. Use of infrared illuminationinto the eye is not objectionable as this light waveband is notperceived by the human eye. The visible or infrared illuminator canactivated or deactivated based on a light level, a distancedetermination, a switch or a control communicably coupled to the visibleor infrared illuminator.

FIG. 7 is a flow chart describing the process for entering a devicesetting and calibration mode from a wireless connected table computer(52), used to acquire an image of the eye from the eye imaging camera(53), transmit the image in a wireless manner to the connected computer(54), check if eye quality is acceptable (55). If the quality isobserved acceptable, the HMDU stores the image in the head-worn deviceas a reference image used to calibrate the eye for tracking (56). Afterthis process is complete, the user can then set and change otherparameters and preferences (57) to modify processing parameters appliedto the image (58) executed on the processor or processors. If the scenequality is acceptable (59), then these settings can be stored intohead-mounted memory (60), and the configuration operation can becompleted (60), or the process of setting parameters can be furthermodified.

For example, the one or more stored settings or preferences can beconfigured by receiving a first message to enter a devicesetting/calibration mode from a remote device, transmitting the firstimage or the modified first image or both the first image and themodified first image to the remote device, receiving a second messagecontaining a change to the one or more stored settings or preferences,implementing the change during one or more of the steps of acquiring thefirst image, modifying the first image and displaying the modified firstimage, transmitting the first image or the modified first image or boththe first image and the modified first image to the remote device,storing the change to the one or more stored settings or preferenceswhenever a third message is received indicating that the first image orthe modified first image or both the first image and the modified firstimage are acceptable, removing the change to the one or more storedsettings or preferences whenever a fourth message is received indicatingthat the first image or the modified first image or both the first imageand the modified first image are not acceptable, and receiving a fifthmessage to exit the device setting/calibration mode from the remotedevice. The steps of receiving the second message, implementing thechange, transmitting the first image or the modified first image or boththe first image and the modified first image, and storing or removingthe change can be repeated until the configuration process is complete.Moreover, the one or more stored settings or preferences can be reset toa set of default settings or preferences.

In another example, the one or more stored settings or preferences canbe configured by receiving a first message to enter a devicesetting/calibration mode from a remote device, transmitting the firstimage or the modified first image or the second image or the modifiedsecond image or a combination thereof to the remote device, storing thesecond image or the modified second image as the stored image of the eyewhenever a sixth message is received indicating that the second image isacceptable, repeating the steps of acquiring and transmitting the firstimage or the modified first image or the second image or the modifiedsecond image or a combination thereof whenever a seventh message isreceived indicating that the second image is not acceptable, andreceiving a fifth message to exit the device setting/calibration modefrom the remote device.

FIG. 8 is a process diagram describing the technique for determining thetypes of image processing algorithms which will be executed on theprocessor or processors, determined by reading sensor information anddetermining the type of scene. The scene is acquired (63) and othersensor settings are read such as light, proximity and motion (63) todetermine the type of task being performed based on this scene activity(64). Image processing algorithms are performed (66) based on this scenedetermination, in combination with pre-stored settings and preferences(69) which were set for the current task (69). The current task mayinclude a close-in reading, a far distance reading, gazing at anexternal electronic display, looking at another person, walking, drivingor other desired task. Finally, the modified image is displayed (67).

FIG. 9 is a process diagram illustrating the computation of focaldistance to an object using a weighed input from two sources (72). Theimage is acquired in combination with an alternative sensor (70) whichcan be used to roughly determine the focal distance to an object. Theimage region used for auto-focus (71) can either be determined from thismeasurement or independently. The current magnification setting andoperating mode (73) also determine the computed focus distance.

FIG. 10 is a process diagram illustrating the use of the eye image tocompute the gaze angle and translate this information to pixel count,which can then be used to offset the image displayed to the eye. Animage of the eye is acquired (78) and the eye gaze angle on two axes(vertical eye gaze angle and horizontal eye gaze angle) is computed (79)by using the stored image of the eye or a preferred locus of a pupil ofthe eye at time of eye tracking calibration (83). Then, the image offsetis computed in degrees (80) and modified based on the currentmagnification setting (82). The image is translated in pixel counts(81). The scene image is acquired (74) and the image is furtherprocessed with image algorithms (75). The scene image is then offset inequal and opposite distance to the computed image offset in degrees (76)and displayed to the eye (77). Note that is some embodiments, thedetermination of the eye gaze angle does not require any initialcalibration or alignment.

FIG. 11 is a process diagram illustrating the use of one or more motionsensors (e.g., a motion measurement sensor to measure an accelerationdata) merged with statistics gathered from a front facing camera toremove excess motion from an image by performing an image offset. Thescene image is acquired (84) and a historical image comparison is used(image subtraction) to estimate the image motion and direction (85)(e.g., an estimated motion data). The results of the historical imagemotion computation are filtered (86). Motion is measured from one ormore motion sensors (91) and the motion data is filtered (92). Both thefiltered motion data (e.g., filtered acceleration data) and filteredhistorical image data (e.g., filtered estimated motion data) are merged(87) and the processor or processors compute an image translation amountin pixels (88), which is further modified by the current magnificationsetting (93) and then used to offset the image (89) and present it tothe eye (90).

FIG. 12 is a process diagram illustrating the use of the eye image tocompute the gaze angle and rate of motion to detect a gestured commandand use this command to change device settings in response to suchcommand. An image of the eye is acquired (94) and the gaze angle andrate of change of the angle is computed (95) based on a stored image ofthe eye at time of eye tracking calibration (99). This information isused to determine eye gestures, which can are interpreted as usercommands (96) by pre-stored settings and preferences (98). The currentdevice settings can then be further modified (97) by responding to theeye gesture command. The scene is captured (100) and image processingalgorithms are performed based on the new user settings and preferences(101) and the image is presented to the eye (102). The image processingalgorithms may include a contrast enhancement algorithm, an edgesharpening algorithm, a virtual real-time aided vision algorithm, anautomatic scene detection and mode setting algorithm, a magnification orimage offset algorithm, an artificial edge highlighting/substitutionalgorithm, a gaze determination algorithm, or any other desiredalgorithm.

FIG. 13 is a process diagram illustrating the use of the eye image tocompute a location of a region of interest (ROI) for applying separateprocessing techniques to an image presented to the eye. An image of theeye is acquired (103) and the eye and is computed (104) using the storedimage of the eye at time of eye tracking calibration (107). The image iscomputed in degrees (105) and a region of interest (110) is computedfrom the current magnification setting (108) and a new region ofinterest is computed (106) which is used to set various parameters inthe processed image, including the auto-focus window, contrastenhancement and region of magnification (111).

FIGS. 14A and 14B are a diagram and process diagram illustrating the useof a series of eye images (115), their computed gaze point (114), angle(113, 116) and magnitude (112) to determine acceleration and location ofa centroid within a smaller region (120) of field of view captured bythe front facing camera (119) to be shifted to a new region (123) withinthe larger image by a proportional movement from an initial point (121)to a final point (122). The first (124) and second (127) images arecaptured and the point of gaze is computed by processor or processors(125). The change in point of gaze (126) is computed and then themagnitude (112) of the change in point of gaze (127) are computed (117and 118) from the resultant angle of gaze (113), and the processor orprocessors shift the image in equal and opposite magnitude (128). Thisfeature is useful in that it compensates for the amplified perceivedmotion in the displayed image, as compared to the scene, whenmagnification is invoked. A simple example of this phenomenon follows.When looking through a telescope with 2× magnification, a 10 degreemovement in the eye results in a 5 degree movement onto the unmagnifiedreal scene. For a given optical field of view (such as 30 degrees in theHMDU), a 2× magnification results in a 15-degree real-scene field ofview, or a 2× reduction in the total information content available onthe microdisplay (albeit at a 2× magnification). Thus, the processdescribed in this Figure of electronically scrolling the displayed imageby an amount proportional to the user's eye movement (e.g., eye gazeangle) and to the magnification setting results in several benefits. Thefirst benefit is that the 30 degree HMDU field of view is retained overthe whole scene, irrespective of the magnification. Referring to theprevious 2× magnification example, as the user's eye moves by 5 degrees,the HMDU electronically shifts the image such that the center of theuser's gaze at 5 degrees matches the real-world, unmagnified scene alsoat 5 degrees. This results in the user's eye movements being morenatural (which is even more important when reading) while stillbenefiting from magnification. The second benefit is that themicrodisplay field of view now matches the scene field of view,irrespective of magnification. This results in less head movements whenmagnification is used, again making the experience closer to a natural(unmagnified) eye and head movements—while still benefiting frommagnification. These angular corrections can be applied to bothhorizontal and vertical eye movements, and at all magnificationsettings.

FIGS. 15A and 15B are a diagram and process diagram describing thecapturing of an image (129) from a front facing camera, shifting aregion of the image at a constant or variable rate (135, 136, 137), andpresenting these sequences of images to the eye (130, 131, 132) tomaximize the perceived field of view of the scene presented to the eye(133). Persons afflicted with low vision, or tunnel vision (such asRetinis Pigmentosa, or RP) tend to scan the scene in front of them, inorder to form a larger image from a smaller view of their restrictedvision, allowing their brain to stitch together a more complete image ofthe scene. The HMDU can perform the same scanning functionelectronically, allowing the user to gaze forward and take-in the samesmaller views of the scene, without the burden of mechanically (usingthe eye, or head, or a combination of the two) moving. This mode may beadjusted to each user's preference (rate of scan, when scanning isinvoked, and other parameters).

FIGS. 16A and 16B are diagrams depicting a configuration in which twomagnification settings are simultaneously applied and presented to theeye such that the image content (139) captured in a frame of the frontfacing camera (138) can be magnified to two different magnificationsettings (140, 141) and presented to the eye. This feature is useful inthat it can provide the user with a larger instantaneous field of viewacross the whole microdisplay which benefiting from a greatermagnification in the center of their gaze (that field of view wouldotherwise be smaller if the whole field of view is magnified to thegreater amount as in the center region). The magnification can beperformed optically using the first camera or electronically using theone or more processors.

FIGS. 17A and 17B are a diagram and flowchart and drawing depicting twoor more colors applied to the foreground and background (147) of animage using one or more image processors. The front facing image iscaptured (142), and the background color of the image is determined bythe processor or processors, and binary thresholding is applied to theimage (143). A new color table (144) is applied to parts of the image(146) and the image is presented to the eye (145). Use of an image thathas been converted and displayed as a binary image improves thecontrast. This is useful for persons with low vision, particularly whenreading. A further step of substituting certain colors onto that binaryimage (for example white to yellow, and black to blue) can further helpin customizing the best sensory abilities of each user's particularpreference or visual medical condition.

FIGS. 18A and 18B are diagrams and FIG. 18C is a flowchart depictingacceleration (151) data measured from one or more motion sensors (153),and front facing camera (152), and one or more processors shifting theimage pixels (155) presented to the eye in equal and opposite magnitudeand direction of the detected motion to mitigate motion blur and shake(150). The displayed field of view without this compensation (149)demonstrates the resulting large movement of the object (148). Thisfeature is useful to help stabilize the magnified image. Since the HMDUis worn on the user's head (either mounted directly or through eyeglassframes), the front camera is subject to the user's slight headmovements. When the image is magnified, slight motion of the head canlead to a complete loss of the viewed context (this is equivalent tolooking through binocular with shaking arms). The HMDU detects, measuresand compensates for that head movement by electronically scrolling themicrodisplay image to counter the effects of the head movements. Thiscompensation also uses the magnification setting as an input.

FIGS. 19A and 19B are diagrams and FIG. 19C is a flowchart depicting themethod of determining the size of text in pixels (158) at a measured ata current focal distance (162), and modifying the magnification (159,161) setting to keep text size constant over a range of focal distancesand text sizes (160). The image is captured from the front facing camera(156) and the focal length (162) to an object being viewed (163) isdetermined in one or more processors (157). This feature is useful forusers who prefer to read while their hands or arms might move back andforth, causing the size of the reading material (in both the scene andthe microdisplay) to change. In this mode of operation, the HMDUelectronically continuously adjusts the magnification up and down asneeded to maintain a constant displayed font size for the user'scomfortable reading experience.

FIGS. 20A and 20B are diagrams and a flowchart depicting the process ofa smartphone or tablet requesting an image from either front facing oreye imaging cameras from the HMDU and the data being transferred anddisplayed on the tablet computer (164) through a wireless link (165)from a wireless transceiver in the HMDU (166). The HMDU is initiallydisconnected (167) until receiving a message from a wireless tabletcomputer to enter into a configuration mode (168), and this message isauthenticated by the HMDU (169). The settings within the HMDU aretransferred in a wireless manner to the tablet computer (170). At thispoint the tablet can disconnect from the HMDU at any time (175), oreither request an image from the device (171) or change settings (174).If an image is requested to be sent, it is transmitted (172) to thetablet for display or further processing (173). This feature is usefulin order to allow a doctor, or the user, or an assistant, to makecertain adjustments to the non-volatile memory of the HDMU whichpersonalize that particular HMDU unit. This also allows the changing andupdating of these features as the user's preferences or needs change.This feature is also useful for transferring information from the HMDHto the tablet, including usage statistics, medical condition diagnosesor indications, images of the user's eye (for calibration, if necessary,of the eye tracking alignment or other diagnostic uses), or otheroperating information of the HMDU, such as temperature, voltages, andother sensor readings.

For example, one or more eye movements can be measured based on thesecond image or the modified second image, and an indication of apotential medical problem can be detected by analyzing the one or moreeye movements. Thereafter, the user can be notified of the indication ofthe potential medical problem or the indication of the potential medicalproblem can be transmitted to a remote device, or the indication of thepotential medical problem can be stored, etc. Statistics can also beperformed and stored based on the measured eye movements.

This feature can also be integrated with a wireless connectivity (suchas high speed cellular data, WiFi, or other generally available wirelessnetworking facilities) to allow for a remote person or computer toassist the user by remotely seeing the user's scene. That remove viewingcan result in mere recording of the scene video (including the displayedvideo or the ET camera video), or a real-time interaction whereby theuser may be assisted in navigating through the current scene. Forexample, a user in a foreign country can use this feature to wirelesslyconnect to a local person who can read signs for them.

FIG. 21 describes the link between the HMDU and the battery pack. Theuser control inputs (183) are encoded in a data packet, and grouped withbattery status (184) and other user-specific data (185) in the batterypack, then modulated (186) and transmitted over the power lead (188)into the HMDU (192) through a demodulator (190) and implemented withother data in the HMDU (191). The return lead (189) is connected to abattery pack feeding the HMDU power (187) and forming a complete circuitwith the battery pack and HMDU. This feature is useful to reduce thenumber of electrical conductors needed to connect the HMDU with thebattery pack. This feature is further useful if the eyeglasses framesare used as the two (and only two) conductors, thus eliminating explicitelectrical wiring from the HMDU to the battery pack, on the frontportion of the glasses. One embodiment of this approach is to attach theHMDU to frames configured to act as two conductors (one for electricalpower and the other one for electrical return). The HMDU directlyattached to the frame for both electrical and mechanical purposes. Thebackside of the eyeglass frames would then (through new electrical wiresthat only emanate from the back of the eyeglass frame stems) to thebattery pack.

Additional features, functionality and elements that can be incorporatedinto the various embodiments of the present invention will not bedescribed.

The placement of the camera and display in a substantially coaxialmanner on the same line as the eye's line-of-sight. The alignment can beperformed electrically, optically, mechanically or a combinationthereof.

The combination of magnification with other image enhancement techniquessimultaneously, such as contrast enhancement, edge sharpening, andothers.

The combination of the above mentioned image enhancement techniques withartificially generated graphical objects, including artificial edgehighlighting, creating a virtual realtime aided vision system.

The ability for the camera to perform automatic focus adjustment for alarge range of scene/gaze distances, while allowing the eye to remain ata single fixed focus distance (set by the corrective glasses and theoptical design of the electronic display system).

The implementation of digital image stabilization in order tocounteracts the effects of head shaking This is particularly useful whenhigh magnification is used, whereby smallest head movements translate toa large perceived image shift, rendering the scene difficult to observe.The amount and dynamic parameters of the stabilization are a function ofthe current magnification setting. The motion data can be collected byone or more sensors.

The implementation of eye tracking, whereby the location and movementsof the aided eye are measured and used for various compensatory schemes.One scheme is to shift the image vertically and/or horizontally in orderto cause the perceived image shift to be equal to the actual scene shiftfor the given angle of the eye's movement, irrespective of the amount ofmagnification currently in effect. Another scheme is to use eyemovements, for example when looking up, as a means of modifying themagnification setting automatically.

The use of a monocular aided system (the device) for a limited centralfield-of-view, with a visual interface to the peripheral view of thesame eye. This forms a seamless visual field of an aided central visionwith a natural (un-aided) peripheral vision.

The use of the above monocular system in conjunction with the un-aidedeye, further causing a seamless binocular view of the scene.

The above but where the image in the aided portion of the aided eye issubstantially modified (e.g., magnification of greater than 1.5, forexample) such that the brain selects which of the images (aided eye withartificial magnification or un-aided eye with unity/naturalmagnification) to use, depending upon the current task-at-hand. This isreferred to as monovision, but with an electronic and adjustable systemof vision enhancements.

The ability of the device to behave differently based on upon thetask-at-hand. This multimode operation (e.g., walking vs. close-inreading, vs. looking at a person's face) can be manually selected by theuser (for example, using a push-button, gesture recognition, speechrecognition). Alternatively, the operating mode of the device can beautomatically set through means such as software analysis of the imagein the current scene, detection of movements through an accelerometer(to detect walking or ambulation), etc.

A method of eye tracking (of imaging the aided eye) in-line with thedisplay through the same prism, or coaxially with the display, sharingsame optical path as the display.

The use of the display as the illumination source and a source ofalignment targets (e.g., fiducials) onto the eye for use by the eyetracking system. This source of illumination or targets can be modulatedin time in fast bursts such that they are not perceived by the user,while synchronizing the eye tracking camera with that source of light.

The use of the eye tracking camera to make measurements of eyemovements, and to analyze these movements in to infer or diagnose thepossibility of certain medical problems, such as the onset of anepileptic seizure, or for similar medical research or diagnosticpurposes.

The use of embedded wireless connectivity (such as with Bluetooth to aSmart Phone) for notification of diagnoses or results from the eyetracking sub-system.

The applicability of eye tracking, image stabilization, and monocularaided vision to the specific problem of helping persons with AMD tobetter see, for a near-eye display system and coaxially-mounted camera.

The ability to electronically scan (shift left/right, up/down)automatically or under user control, allowing persons with tunnel vision(e.g., due to Glaucoma or Retinitis Pigmentosa) to see a large field ofview over time than their natural eye allows, without having to movetheir eyes or head (or as much). This is possible due to a larger fieldof view of the camera as compared to the display and/or of the damagedeye's remaining central vision.

The implementation of vision tests that are normally administered usingexternal visual targets, such the Amsler chart or the “eye chart” (toestablish the visual acuity) by using the internal electronic displayand electronically generated images, rather than images of wallmountedcharts.

The ability to generate electronically pre-programmed sequences ofimages (from memory, and/or from a wirelessly connected device, such asa Smart Phone) to help exercise the eye (e.g., as used for baseballbatters to improve eye speed response).

The mounting of the battery pack and user controls on the neck, ratherthan on the head or other places on the body, allowing for easy accessto controls while also managing the weight of the batteries in a singleconvenient location.

The ability for the device to be configured by the doctor and by thepatient for preferences, such as default and minimum/maximum values formagnification, contrast enhancement, artificial edge enhancements, andother image enhancing algorithm settings. These settings are stored innon-volatile memory within the head-mounted system, but are accessedwirelessly (e.g., Bluetooth), through a software application executingon a Smart Phone.

The ability to acquire, examine, and to select/deselect eye trackingcalibration images. The head-mounted display, after being commanded bythe Smart Phone (wirelessly) takes an image of the eye and transmits itto be viewed by the doctor. The doctor decides if the image isacceptable, and if the patient was indeed gazing at the proper angle,and commands the storage of that image as the reference eye trackingimage from which all other eye tracking angular computations aresubsequently made. Note that the eye need not be gazing forward, as thedoctor can accept the angle and enter it as an offset to the subsequentcomputations of eye tracking

The ability to customize (and to store in non-volatile memory) userpreferences, including eye gesture recognition commands (such asmagnification change and amount of magnification based on the eyesweeping upwards, the rate of sweep, etc.)

The ability for the head-mounted display system to act as a wirelessdisplay monitor for displaying the screen that would be shown on anexternal device, such as a Smart Phone.

Rather than looking at the Smart Phone display (through the head-mountedsystem's camera, image processor and microdisplay), the data that ispresented on the screen may be wirelessly transmitted to thehead-mounted system for direct display (through the internal imageprocessor) to the microdisplay, thus bypassing the camera.

Each of these operations/features/functions listed below may beimplemented independently or as a combination with other features. Insome cases, one feature enables other features to be implemented, whichwould not be able to be done without the enabling feature (for example:eye tracking enables field-of-view compensation by shifting the imagebased on magnification and current point-of-gaze).

Change magnification with eye tracking in vertical direction (likebifocals or progressive lenses which vary the setting based on theheight of the gaze).

Auto-focus using image analysis, and/or using a secondary sensor (e.g.,a laser range finder), and a combination of the two based on the rangeof interest (e.g., laser for short range, image analysis long range)

Auto-focus using a third camera set to a different focus range, or acomplex lens that sets the focus in different parts of the image atseparate unique focus distances (all for faster finding the focuslocation).

Gesture recognition or eye control or implement commands (enabled by eyetracking) (e.g., “click” with blinking eye, etc.).

Automatic control of operation by scene analysis and detection.

Reading (invokes color substitution for better contrast) vs. walking(invokes artificial edge enhancement).

Automatic turn-on of flashlight (visible or IR) for low light close-inreading based on light level and auto-focus distance determination(which can be via scene analysis or a secondary distance sensor).

Automatic magnification setting when reading to keep the displayed textat a particular size (based on the user's vision and preference)irrespective of the text size in the scene.

The device can go to sleep (low power mode) when user removes devicefrom head (use of a proximity sensor, IR), and wake-up (normal powermode) in the opposite case. Similarly, the device can go to sleep (lowpower mode) when user goes to sleep (does not move head for a while),and wake-up (normal power mode) in the opposite case. The second imageof the eye can be used to detect that the eye is closed for a specifiedperiod of time, or is open after being closed for the specified periodof time.

Image stabilization (depending on magnification setting) usingaccelerometer (or image analysis).

Means of determining the point of gaze (second camera, optics,algorithms—electronically)

Means of determining the point of gaze using optical correlationtechniques (http://www.grc.nasa.gov/WWW/OptInstr/Wernet_Web/SPOF.html)

No-initial calibration or alignment needed for eye tracking, using fixedproperly fitted glasses and system as knowing where the “center of theworld” is located.

The same optical path can be used for the eye-tracking camera as is usedfor the display (to save overall size).

The image display can be used as a source of illumination for theeye-tracking camera (occasionally flash a pre-determined image from thedisplay to the eye, in synchronization with the eye-tracking camera whocan use this to make an eye measurement, or have a border on the imagealways, etc.). One or more illumination devices may also be configuredto face towards the eye.

The point of gaze can be used for scrolling image based on magnification(for effective live-scene FOV while maintaining magnification).Electronic image scrolling adjustments (if not all three items aremechanically in-line) can be based on magnification and/or point of gazeand/or auto-focus distance determination.

The point of gaze can also be used set the auto-focus zone of interest(very useful when reading a book which is tilted relative to the viewingangle, where the top of the page and the bottom of the page are at asignificantly different distance to the eye, and only the eye is movingto read it all).

The point of gaze can be used to set the image quality parameters(color, black level, white level, gain, gamma). For example,region-of-interest increased contrast (center region of macular area ondisplay, with progressive change from rest of “peripheral” un-enhancedimage).

The point of gaze zone size can be variable as a function of currentmagnification setting.

Establish line-of-sight of camera, and display and mounting onto lensesall concentric. Do the above using mechanical alignment of all threeelements.

Purposely add an offset to that line-of-sight to compensate for theindividual user's preferred retina locus or eccentric viewing.

Electronically scan in a pre-determined pattern (e.g., left-to-right,etc.) of the real scene onto the display to emulate a person's scanningusing their eye or head movements (especially for people with tunnelvision).

No frame buffer, no DRAM, use of RGB pixels in display (minimal latencyof image). Use of RGB pixels (vs. color progressive display) and otherdigital circuitry allows the no-use of a frame buffer. In other words,there is substantially no propagation delay (e.g., less than aboutone-tenth frame delay) between acquiring the first image and displayingthe modified first image.

Combination of digital image processing and LCOS (LiquidCrystal-on-Silicon) or OLED (Organic LED) electronic display on the samedie, saving space and power.

Orientation of overall mechanical design to be vertically longer tominimize sideways encroachment, to maximize sideways peripheral vision(also, raise it higher vs. lower to maximize bottom-side peripheralvision).

Data over power for reduced wires from battery pack to head-mountedunit.

Use of glasses frames as electrical conductors (e.g., power and groundon each half-frame) to make it wireless in the front and/or heat sinks.

Use of two separate monocular systems (each with its own camera, imageprocessing and display) but coordinate the two for a binocular system bycommunicating system settings in real-time (e.g., focus setting, cameragain, magnification, etc.) to ensure that both eyes operate together,but yet each has his own complete and otherwise independent hardware.

Ability for the device described herein to further allow forcoordination of both eyes (such as focus distance) but to perform othercorrections (color, brightness, contrast, etc.) uniquely for each eye tomaximize the overall perceived image quality for each individual user.The brightness may be adjusted based on a medical diagnosis, an eyesensitivity or a background illumination.

Ability to wirelessly transmit video (either or both scene image, eyetracking image, or combinations thereof) to an outside device for bettertesting, monitoring, etc.

Ability to wirelessly transmit video of the scene to allow a third partyperson to also see the scene to help the user understand the scene(e.g., a form of “facetime” where the camera is what's mounted on theuser's glasses). Also, doctor might be able to look at the patient's eyeremotely to help with diagnoses (“eyetime”).

Combination of the device described herein specifically with a diseasedeye (e.g., damaged retina, AMD, RP, etc.).

Combination of the device described herein with the use of an implantedlens (IMT or Cataract, or even LASIC) to complement the optical designof the display optics (making overall size smaller). Similarly,combination of the device described herein with the use of an implantedartificial retina to complement the overall functionality of the newartificial eye. The device is configured to complement, coordinate orcommunicate with the implant or artificial eye.

Perform visual acuity and other tests (e.g., Amsler chart) usingdisplay, record and report the results (wireless connectivity within thedevice)

Measure eye movements in normal operation, gather statistics, analyzeand communicate to help in the diagnosis of various medical conditions,such as neurological problems (Traumatic Brain Injury, Parkinson's,epileptic seizures, etc.). Based this analysis, different displaypatterns can be implemented to help reduce eye strain or “relax” theperson.

Use eye-tracking camera and optics to look into the retina for furtherdiagnosis of evolving macular degeneration.

A single head- or glasses-mounted system that includes the placement ofthe camera and display (in a coaxial manner, or electronically correctedto be coaxial) on the same line as the eye's line-of-sight, thecombination of magnification with other image enhancement techniquessimultaneously, such as contrast enhancement, edge sharpening,artificial edge highlighting, and others, the combination of the abovementioned image enhancement techniques with artificially-generatedgraphical objects, including artificial edge highlighting, creating avirtual real-time aided vision system, and the ability for the camera toperform automatic focus adjustment for a large range of scene/gazedistances, while allowing the eye to remain at a single fixed focusdistance (set by the corrective glasses and the optical design of theelectronic display system).

The implementation of digital image stabilization in order tocounteracts the effects of head shaking This is particularly useful whenhigh magnification is used, whereby smallest head movements translate toa large perceived image shift, rendering the scene difficult to observe.The amount and dynamic parameters of the stabilization are a function ofthe current magnification setting.

The implementation of eye tracking, whereby the location and movementsof the aided eye are measured and used for various compensatory schemes,such as to shift the image vertically and/or horizontally in order tocause the perceived image shift to be equal to the actual scene shiftfor the given angle of the eye's movement, irrespective of the amount ofmagnification currently in effect, or to use eye movements, for examplewhen looking up, as a means of modifying the magnification settingautomatically.

The use of a monocular aided system for a limited central field-of-view,with a visual interface to the peripheral view of the same eye. Thisforms a seamless visual field of an aided central vision with a natural(un-aided) peripheral vision.

The use of the above monocular system in conjunction with the un-aidedeye, further causing a seamless binocular view of the scene.

The image in the aided portion of the aided eye is substantiallymodified (e.g., magnification of greater than 1.5, for example) suchthat the brain selects which of the images (aided eye with artificialmagnification or un-aided eye with unity/natural magnification) to use,depending upon the current task-at-hand. This is referred to asmono-vision, but with an electronic and adjustable system of visionenhancements.

The ability to behave differently based on upon the task-at-hand. Thismulti-mode operation (e.g., walking vs. close-in reading, vs. looking ata person's face) can be manually selected by the user (for example,using a push-button, gesture recognition, speech recognition).Alternatively, the operating mode can be automatically set through meanssuch as software analysis of the image in the current scene, detectionof movements through an accelerometer (to detect walking or ambulation),etc.

A method of eye tracking (of imaging the aided eye) in-line with thedisplay through the same prism, or coaxially with the display, sharingsame optical path as the display.

The use of the display as the illumination source and a source ofalignment targets (e.g., fiducials) onto the eye for use by the eyetracking system. This source of illumination or targets can be modulatedin time in fast bursts such that they are not perceived by the user,while synchronizing the eye tracking camera with that source of light.

The use of the eye tracking camera to make measurements of eyemovements, and to analyze these movements in to infer or diagnose thepossibility of certain medical problems, such as the onset of anepileptic seizure, or for similar medical research or diagnosticpurposes.

The use of embedded wireless connectivity (such as with Bluetooth to aSmart Phone) for notification of diagnoses or results from the eyetracking sub-system.

The applicability of eye tracking, image stabilization, and monocularaided vision to the specific problem of helping persons with AMD tobetter see, for a near-eye display system and coaxially-mounted camera.

The ability to electronically scan (shift left/right, up/down)automatically or under user control, allowing persons with tunnel vision(e.g., due to Glaucoma or Retinitis Pigmentosa) to see a large field ofview over time than their natural eye allows, without having to movetheir eyes or head (or as much). This is possible due to a larger fieldof view of the camera as compared to the display and/or of the damagedeye's remaining central vision.

The implementation of vision tests that are normally administered usingexternal visual targets, such the Amsler chart or the “eye chart” (toestablish the visual acuity) by using the internal electronic displayand electronically generated images, rather than images of wall-mountedcharts.

The ability to generate electronically pre-programmed sequences ofimages (from memory, and/or from a wirelessly connected device, such asa Smart Phone) to help exercise the eye (e.g., as used for baseballbatters to improve eye speed response)—this could be monocular orbinocular or bi-monocular (one eye at a time).

The mounting of the battery pack and user controls on the neck, ratherthan on the head or other places on the body, allowing for easy accessto controls while also managing the weight of the batteries in a singleconvenient location.

The ability for the device to be configured by the doctor and by thepatient for preferences, such as default and minimum/maximum values formagnification, contrast enhancement, artificial edge highlighting, andother image enhancing algorithm settings.

These settings are stored in non-volatile memory within the head-mountedsystem, but are accessed wirelessly (e.g., Bluetooth, WiFi), through asoftware application executing on a wireless device accessing theInternet.

The ability to acquire, examine, and to select/deselect eye trackingcalibration images. The head-mounted display, after being commanded bythe Smart Phone (wirelessly) takes an image of the eye and transmits itto be viewed by the doctor. The doctor decides if the image isacceptable, and if the patient was indeed gazing at the proper angle,and commands the storage of that image as the reference eye trackingimage from which all other eye tracking angular computations aresubsequently made. Note that the eye need not be gazing forward, as thedoctor can accept the angle and enter it as an offset to the subsequentcomputations of eye tracking.

The ability to customize (and to store in non-volatile memory) userpreferences, including eye gesture recognition commands (such asmagnification change and amount of magnification based on the eyesweeping upwards, the rate of sweep, etc.)

The ability for the head-mounted display system to act as a wirelessdisplay monitor for displaying the screen that would be shown on anexternal device, such as a Smart Phone. Rather than looking at the SmartPhone display (through the head-mounted system's camera, image processorand microdisplay), the data that is presented on the screen may bewirelessly transmitted to the head-mounted system for direct display(through the internal image processor) to the microdisplay, thusbypassing the camera.

The first camera may include an automatic focusing device. The secondcamera may a fixed focusing device or an automatic focusing device. Themicrodisplay may include an optical magnifier. The one or moreprocessors and the microdisplay can be integrated into a singlesemiconductor die.

The microdisplay can be defined by a first zone and a second zone,wherein the first zone is a whole region of the microdisplay magnifiedby a background magnification amount, and the second zone is acontiguous zone within the first zone magnified by a differentmagnification amount. A center location of the second zone within thefirst zone can be computed from the gaze angle.

In addition, a time sequence of the first images or the modified firstimages or both the first images and the modified first images can betransmitted to an external device. Note that in some cases the imagesmay also include the second image or the modified second image. Theexternal device can store or view or process the time sequence of thefirst images or the modified first images or both the first images andthe modified first images. The device can receive information from theexternal device based on the time sequence. Moreover, the device canreceive a fourth image or a sequence of images or an information fromthe external device, create a fifth image by processing the receivedfourth image, or the sequence of images or the information using the oneor more processors, and displaying the fifth image on the microdisplay.

This application claims priority to U.S. provisional patent applicationSer. No. 61/941,777 filed on Feb. 19, 2014 and entitled “Apparatus toassist with low vision” the contents of which are hereby incorporated byreference in their entirety.

It will be understood that particular embodiments described herein areshown by way of illustration and not as limitations of the invention.The principal features of this invention can be employed in variousembodiments without departing from the scope of the invention. Thoseskilled in the art will recognize, or be able to ascertain using no morethan routine experimentation, numerous equivalents to the specificprocedures described herein. Such equivalents are considered to bewithin the scope of this invention and are covered by the claims.

All publications, patents and patent applications mentioned in thespecification are indicative of the level of skill of those skilled inthe art to which this invention pertains. All publications and patentapplications are herein incorporated by reference to the same extent asif each individual publication or patent application was specificallyand individually indicated to be incorporated by reference.

The use of the word “a” or “an” when used in conjunction with the term“comprising” in the claims and/or the specification may mean “one,” butit is also consistent with the meaning of “one or more,” “at least one,”and “one or more than one.” The use of the term “or” in the claims isused to mean “and/or” unless explicitly indicated to refer toalternatives only or the alternatives are mutually exclusive, althoughthe disclosure supports a definition that refers to only alternativesand “and/or.” Throughout this application, the term “about” is used toindicate that a value includes the inherent variation of error for thedevice, the method being employed to determine the value, or thevariation that exists among the study subjects.

As used in this specification and claim(s), the words “comprising” (andany form of comprising, such as “comprise” and “comprises”), “having”(and any form of having, such as “have” and “has”), “including” (and anyform of including, such as “includes” and “include”) or “containing”(and any form of containing, such as “contains” and “contain”) areinclusive or open-ended and do not exclude additional, unrecitedelements or method steps.

The term “or combinations thereof” as used herein refers to allpermutations and combinations of the listed items preceding the term.For example, “A, B, C, or combinations thereof” is intended to includeat least one of: A, B, C, AB, AC, BC, or ABC, and if order is importantin a particular context, also BA, CA, CB, CBA, BCA, ACB, BAC, or CAB.Continuing with this example, expressly included are combinations thatcontain repeats of one or more item or term, such as BB, AAA, AB, BBC,AAABCCCC, CBBAAA, CABABB, and so forth. The skilled artisan willunderstand that typically there is no limit on the number of items orterms in any combination, unless otherwise apparent from the context

Although the present invention and its advantages have been described indetail, it should be understood that various changes, substitutions andalterations can be made herein without departing from the spirit andscope of the invention as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods and steps described in the specification, butonly by the claims.

The invention claimed is:
 1. A computerized method for improving,augmenting or enhancing a vision of a person, comprising the steps of:providing a wearable apparatus proximate to an eye of the person suchthat a second eye of the person is unobstructed, the wearable apparatuscomprising a first camera configured to acquire a first image of a scenefacing away from the eye, a second camera configured to acquire a secondimage of the eye, one or more sensors, a microdisplay configured todisplay a modified first image to the eye, and one or more processorscommunicably coupled to the first camera, the second camera, the one ormore sensors and the microdisplay, and wherein the wearable apparatus issized to maintain a peripheral vision of the first eye, and the one ormore sensors comprise one or more of a motion sensor, a temperaturesensor, an ambient light detector, a rangefinder, a proximity sensor andan infrared sensor; acquiring the first image of the scene using thefirst camera; acquiring the second image of the eye using the secondcamera; modifying the second image using the one or more processors;determining an eye gaze angle based on the second image or the modifiedsecond image using the one or more processors; modifying the first imagebased on one or more vision improvement parameters and a data from theone or more sensors, and offsetting the first image by an image offsetbased the eye gaze angle using the one or more processors; anddisplaying the modified first image on the microdisplay to improve,augment or enhance the vision of the person.
 2. The computerized methodas recited in claim 1, further comprising the step of occasionallyflashing a predetermined image on the microdisplay.
 3. The computerizedmethod as recited in claim 1, wherein the step of displaying themodified first image on the microdisplay further comprises the step ofscanning the modified first image onto the microdisplay in apredetermined pattern to emulate the person's eye or head movements. 4.The computerized method as recited in claim 1, wherein the step ofmodifying the first image based on the one or more vision improvementparameters comprises the step of: offsetting the first image based on apreferred retina locus or an eccentric viewing of the person; enhancinga contrast of the first image; or adjusting a brightness of the firstimage based on a medical diagnosis, an eye sensitivity, or a backgroundillumination.
 5. The computerized method as recited in claim 1, furthercomprising the step of magnifying the first image of the scene based ona current magnification selected from two or more magnification settingsusing the first camera or the one or more processors, wherein thecurrent magnification magnifies a whole field of view or only a regioncentered on a point of gaze.
 6. The computerized method as recited inclaim 1, further comprising the step of automatically focusing the firstimage using: an image analysis based on the eye gaze angle, the one ormore sensors or a combination thereof, wherein the one or more sensorscomprise a range finder; a third camera configured to acquire a thirdimage of the scene facing away from the eye that is set to a differentfocus range than the first camera; or a complex lens optically connectedto the third camera that sets a focus in different parts of the thirdimage at separate unique focus distance.
 7. The computerized method asrecited in claim 1, wherein the image offset is further based on animage stabilization comprising the steps of: measuring a motion datacomprising an acceleration data using the one or more sensors, whereinthe one or more sensors comprise a motion measurement sensor;determining an estimated motion data by comparing the first image of thescene to one or more previous first images of the scene; and determiningthe image stabilization by merging the acceleration data with theestimated motion data.
 8. The computerized method as recited in claim 1,wherein the step of modifying the first image based on the one or morevision improvement parameters comprises the step of enhancing the firstimage of the scene using one or more image processing algorithmsselected from at least one of a contrast enhancement algorithm, an edgesharpening algorithm, a virtual real-time aided vision algorithm, anautomatic scene detection and mode setting algorithm, a magnification orimage offset algorithm, an artificial edge highlighting/substitutionalgorithm or a gaze determination algorithm.
 9. The computerized methodas recited in claim 8, wherein the automatic scene detection and modesetting algorithm comprises the steps of: determining a scene type and acurrent task by analyzing the first image, the data from the one or moresensors, or a combination thereof; changing the one or more visionimprovement parameters to match the scene type and the current task orone or more stored settings or preferences, or a combination thereof;and wherein the current task comprises a close-in reading, a fardistance reading, gazing at an external electronic display, looking atanother person, walking or driving.
 10. The computerized method asrecited in claim 1, wherein the step of modifying the first image basedon the one or more vision improvement parameters further comprises thestep of enhancing the first image of the scene using an eye gesturerecognition and mode setting algorithm comprising the steps of:determining an eye gaze rate of change; determining a direction of aneye gaze motion; determining an eye gesture command based on the eyegaze angle, the eye gaze rate of change and the direction of the eyegaze motion; and changing the one or more vision improvement parametersor magnification in response to the eye gesture command based on one ormore stored settings or preferences.
 11. The computerized method asrecited in claim 1, further comprising the step of activating ordeactivating a visible or infrared illuminator based on a light level ora distance determination, wherein the visible or infrared illuminator isconfigured to face towards the scene and is communicably coupled to theone or more processors.
 12. The computerized method as recited in claim1, wherein the step of modifying the first image based on the one ormore vision improvement parameters further comprises the step ofmaintaining a size of a text within the modified first image at aspecified size irrespective of an actual size of the text within thefirst image.
 13. The computerized method as recited in claim 1, furthercomprising the steps of: entering a low power mode whenever the one ormore sensors detects the person removing the wearable apparatus or goingto sleep or the second image indicates that the eye is closed for aspecified period of time; and entering a normal power mode whenever theone or more sensors detects the person putting the wearable apparatus onor awakening from sleep, the second image indicates that the eye is openafter being closed for the specified period of time.
 14. Thecomputerized method as recited in claim 1, further comprising the stepof configuring one or more stored settings or preferences by: receivinga first message to enter a device setting/calibration mode from a remotedevice; transmitting the first image or the modified first image or boththe first image and the modified first image to the remote device;receiving a second message containing a change to the one or more storedsettings or preferences; implementing the change during one or more ofthe steps of acquiring the first image, modifying the first image anddisplaying the modified first image; transmitting the first image or themodified first image or both the first image and the modified firstimage to the remote device; storing the change to the one or more storedsettings or preferences whenever a third message is received indicatingthat the first image or the modified first image or both the first imageand the modified first image are acceptable; removing the change to theone or more stored settings or preferences whenever a fourth message isreceived indicating that the first image or the modified first image orboth the first image and the modified first image are not acceptable;and receiving a fifth message to exit the device setting/calibrationmode from the remote device.
 15. The computerized method as recited inclaim 1, further comprising the step of configuring one or more storedsettings or preferences by: receiving a first message to enter a devicesetting/calibration mode from a remote device; transmitting the firstimage or the modified first image or the second image or the modifiedsecond image or a combination thereof to the remote device; storing thesecond image or the modified second image as the stored image of the eyewhenever a sixth message is received indicating that the second image isacceptable; repeating the steps of acquiring and transmitting the firstimage or the modified first image or the second image or the modifiedsecond image or a combination thereof whenever a seventh message isreceived indicating that the second image is not acceptable; andreceiving a fifth message to exit the device setting/calibration modefrom the remote device.
 16. The computerized method as recited in claim1, further comprising the steps of: measuring one or more eye movementsbased on the second image or the modified second image; detecting anindication of a potential medical problem by analyzing the one or moreeye movements; and notifying the user of the indication of the potentialmedical problem or transmitting the indication of the potential medicalproblem to a remote device, or storing the indication of the potentialmedical problem.
 17. The computerized method as recited in claim 1,further comprising the step of: performing an eye test by inserting aneye test chart into the modified first image; performing an eye exerciseby inserting a pre-programmed sequence of images into the modified firstimage; or inserting a pre-programmed sequence of images into themodified first image to reduce a strain of the eye or to relax theperson.
 18. The computerized method as recited in claim 1, wherein: thefirst image and the modified first image are not buffered; or there issubstantially no propagation delay between acquiring the first image anddisplaying the modified first image.
 19. The computerized method asrecited in claim 1, further comprising the step of transmitting a timesequence of the first images or the second images or the modified firstimages or the modified second images or any combination thereof to anexternal device.
 20. The computerized method as recited in claim 1,further comprising the steps of: receiving a fourth image or a sequenceof images or an information from an external device; creating a fifthimage by processing the received fourth image, or the sequence of imagesor the information using the one or more processors; and displaying thesixth image on the microdisplay.
 21. A wearable apparatus for improving,augmenting or enhancing a vision of a person, comprising: a first cameraconfigured to acquire a first image of a scene facing away from an eyeof the person; a second camera configured to acquire a second image ofthe eye; one or more sensors comprising one or more of a motion sensor,a temperature sensor, an ambient light detector, a rangefinder, aproximity sensor and an infrared sensor; a microdisplay configured todisplay a modified first image to the eye such that a second eye of theperson is unobstructed; one or more processors communicably coupled tothe first camera, the second camera, the one or more sensors and themicrodisplay, wherein the one or more processors are configured toacquire the first image of the scene using the first camera, acquire thesecond image of the eye using the second camera, modify the secondimage, determining an eye gaze angle based on the second image or themodified second image, modify the first image based on one or morevision improvement parameters and a data from the one or more sensors,and by offsetting the first image using an the image offset based on theeye gaze angle, and display the modified first image on the microdisplayto improve, augment or enhance the vision of the person; and thewearable apparatus is sized to maintain a peripheral vision of the firsteye.
 22. The wearable apparatus as recited in claim 21, furthercomprising a visible or infrared illuminator communicably coupled to theone or more processors and configured to face towards the scene, whereinthe visible or infrared illuminator is activated or deactivated based ona light level or a distance determination.
 23. The wearable apparatus asrecited in claim 21, wherein the wearable apparatus is attached to orintegrated into a monocle, a pair of glasses, a pair of sunglasses or aframe to support the apparatus.
 24. The wearable apparatus as recited inclaim 23, wherein the frame for the pair of glasses or sunglassesprovides a heat sink for the apparatus.
 25. The wearable apparatus asrecited in claim 23, further comprising a second apparatus attached toor integrated into the pair of glasses, the pair of sunglasses or theframe, wherein the wearable apparatus and the second apparatuscommunicate with one another.
 26. The wearable apparatus as recited inclaim 21, wherein the wearable apparatus is mounted onto a frame infront of the eye or a pupil of the eye.
 27. The wearable apparatus asrecited in claim 21, wherein the wearable apparatus further comprisesone or more controls or one or more batteries.
 28. The wearableapparatus as recited in claim 21, wherein the first camera and themicrodisplay are substantially coaxially aligned with the eyeelectrically, optically, mechanically or a combination thereof.
 29. Thewearable apparatus as recited in claim 21, further comprising one ormore controls communicably coupled to the one or more processors whereinthe one or more controls comprise a knob, a button, a capsense, aslider, a voice activated control, an eye motion activated control, afocus control, a gesture recognition control, an automatic sleep/wake-upcontrol, or a combination thereof.
 30. The wearable apparatus as recitedin claim 21, further comprising an automatic focusing devicecommunicably coupled to the first camera.
 31. The wearable apparatus asrecited in claim 21, wherein the one or more processors and themicrodisplay are integrated into a single semiconductor die.
 32. Thewearable apparatus as recited in claim 21, wherein: the wearableapparatus further comprises a control unit that is communicably coupledto the wearable apparatus wirelessly or via one or more conductors; thecontrol unit comprises one or more status indicators, one or morecontrols, one or more batteries and a battery charger electricallyconnected to the one or more batteries, or a combination thereof. 33.The wearable apparatus as recited in claim 21, wherein the wearableapparatus is configured to complement, coordinate or communicate with animplant within the eye or the eye comprises an artificial eye.
 34. Thewearable apparatus as recited in claim 21, wherein: the microdisplay isdefined by a first zone and a second zone; the first zone comprises awhole region of the microdisplay magnified by a background magnificationamount; and the second zone comprises a contiguous zone within the firstzone magnified by a different magnification amount.
 35. The wearableapparatus as recited in claim 21, further comprising an external devicecommunicably coupled to the one or more processors, wherein the one ormore processors are further configured to transmit a time sequence ofthe first images or the second images or the modified first images orthe modified second images or any combination thereof to an externaldevice.
 36. The wearable apparatus as recited in claim 21, furthercomprising an external device communicably coupled to the one or moreprocessors, wherein the one or more processors are further configuredto: receive a fourth image or a sequence of images or an informationfrom the external device; create a fifth image by processing thereceived fourth image, or the sequence of images or the information; anddisplaying the sixth image on the microdisplay.